Open source, standard tooling for experimental protocols: towards Registered reports
Old school:
What if… → Garden of forking paths (Rubin 2017)
p-hacking, hypothesizing after the results are known (i.e. HARKing)
The need for significance and novelty
False positives research
Open-Science-Collaboration (2015)
Baker (2016)
Improve methods: increase n, better measures and manipulations, improve design, piloting
Reduce temptations: p-hacking, hypothesizing after the results are known (i.e. HARKing), selective reporting → preregistration, internal replications
Openness: transparency of the research process, sharing methods, materials, procedures, and data
Nosek et al. (2022)
RRs were conceived to alter the incentives for authors and journals away from producing novel, positive, clean findings and towards conducting and publishing rigorous research on important questions. Soderberg et al. (2021)
More open, preregistered, reproducible by default
It does not matter if p value is < 0.05
Less incentives for p-hacking
More trustworthy results
Before having the data available, it is hard to know how to analyze it
There are always surprises when receiving new data. How can I create an analysis plan that will hold?
We (CSCN; ~5-10 PI’s) used different technologies to develop experiments: Psychopy, Qualtrics, Limesurvey, jsPsych, etc.
Each of these has advantages and disadvantages.
Mostly, pragmatic aspects guided the decision: lab history and resources, coding experience, type of experiment (EEG/behavioral, lab/online), …
Each protocol started almost from scratch. Sometimes a single task would define the technology used.
At some point, we had multiple implementations of the same tasks in different technologies, not always exact replicas.
Some would work in certain computers, other did not. Output data wildly different.
2 questions voluntary survey:
Open source tools to help create experimental paradigms with jsPsych, simulate participants and standardize the data preparation and analysis
1) jsPsychMaker
A big catalog of tasks in jsPsychMaker. Each task runs with jsPsychMonkeys to create virtual participants, and have a script in jsPsychHelpeR to automate data preparation (re-coding, reversing items, calculating dimensions, etc.).
The final goal is to help you have the data preparation and analysis ready before collecting any real data, reducing errors in the protocols, and making the move towards registered reports easier.
Current developers:
Initial idea and development:
Discussions, ideas, testing:
Creating new tasks is as simple as:
Copy paste items, adapt csv/excel file
Create protocol (see next slide)
Create a protocol with three existing tasks plus the adapted Example tasks:
Release a single Monkey and take a look:
Release 10 Monkeys in parallel:
Create project for data preparation:
Create a task correction script:
Visualize and run data preparation:
Create protocol, simulate participants and prepare data…
Let’s try to download the data, process it and show a report with the results:
Plan A: run Experiment Issues project
Plan B: If something fails, we always have the monkeys!
Very easy to create new scales, and simple tasks, but complex experimental tasks require javascript and HTML knowledge (although there are a good number of examples available)
Data preparation for new experimental tasks requires expertise in R (simple surveys not so much)
Analysis reports require some R knowledge (simple templates available)
Requires access to a server for online tasks
Only behavioral tasks
Javascript programmers
R programmers
Task creators
Testers
With jsPsychR, protocols are standardized and with (mostly) clean code
Data preparation is 90% automatic, standardized, and beautiful
When errors are found and fixed, old protocols can benefit from the corrections, old results can be checked, …
Super easy to work on analysis before collecting human data
Much easier to write up a good analysis plan, share it, improve it, …
Sharing protocol, materials, data preparation is trivial (single command)
Creating future proof full reproducible projects (with Docker) is one command away
For more information about RR, templates, checklists, list of participating journals (>300), etc.:
Also, check out the future:
And our manual:
Gorka Navarrete
gorkang@gmail.com
https://fosstodon.org/@gorkang